System and Method for Media Synchronization and Collaboration

ABSTRACT

An improved system and method for media synchronization and collaboration involves a data storage, a plurality of media recording devices used by a plurality of users to independently record an event from multiple locations thereby producing a plurality of recorded media data corresponding to a plurality of views of the event, and a media player comprising a processor and a graphical user interface. Each of the plurality of media recording devices convey to the data storage media data and metadata corresponding to their respective view of the event, where the metadata includes time samples in accordance with a common time reference. The media player uses the metadata to synchronize and play the plurality of views of the event. The graphical user interfaces can be used to select views of the plurality of views of the event to be playing during periods of time as part of an overall timeline of a composite video that consists of a sequence of selected views. The media player is configured to create a multiple view video data package comprising a plurality of media files and metadata files corresponding to multiple views of a recorded event. The multiple view video data package may include a composite video produced using the plurality of media files and metadata files. The media player can be configured to post the multiple view video data package to an internet media-sharing website thereby enabling users of the internet media-sharing website to at least one of comment on the composite video, rate the composite video, or download the plurality of media files and the plurality of metadata files.

CROSS-REFERENCE TO RELATED APPLICATIONS

This United States Non-provisional application is a continuation-in-partof U.S. patent application Ser. No. 14/988,568, filed Jan. 5, 2016,which claims the benefit of U.S. Provisional Patent Application No. U.S.62/099,937, filed Jan. 5, 2015, titled “A System and Method forCloud-based Media Streaming”.

This U.S. Non-provisional patent application also claims the benefit ofU.S. Provisional Patent Application No. U.S. 62/340,013, filed May 23,2016, titled “System and Method for Cloud-based Media Synchronizationand Collaboration”.

These patent applications are each incorporated by reference herein intheir entirety.

FIELD OF THE INVENTION

The present invention relates generally to a system and method for mediasynchronization and collaboration. More particularly, the presentinvention relates to a system and method for media synchronization andcollaboration, where metadata is used to synchronize media allowingmultiple views of an event recorded independently by multiple mediarecording devices to be synchronized and combined into a collaborativemedia file.

SUMMARY OF THE INVENTION

Briefly, one aspect of the present invention involves an improved systemfor media synchronization and collaboration that includes a datastorage, a plurality of media recording devices used by a plurality ofusers to independently record an event from multiple locations therebyproducing a plurality of recorded media data corresponding to aplurality of views of the event, each of the plurality of mediarecording devices conveying to the data storage media data and metadatacorresponding to their respective view of the event, the metadataincluding time samples in accordance with a common time reference, and amedia player comprising a processor and a graphical user interface, themedia player using the metadata to synchronize and play the plurality ofviews of the event.

The plurality of media recording devices can be a plurality of smartphones.

The graphical user interface may include a plurality of viewing windowsused to display the plurality of views of the event.

Each of the plurality of viewing windows can have a correspondingselection icon of a plurality of selection icons.

The plurality of selection icons can be color-coded.

The plurality of selection icons can be used to select views of theplurality of views of the event to be playing during periods of time aspart of an overall timeline of a composite video that consists of asequence of selected views.

The graphical user interface can play the composite video.

The timeline can be color-coded to identify the view of the plurality ofviews that is playing during a given period of time of the timeline.

The graphical user interface can include a multi-view combinationcontrol that causes a sequence of different views of the same timeperiod or overlapping time periods to be played in the composite video.

The metadata can include location and view angle data and the graphicaluser interface can provide the ability to display a location/view angleindicator window that indicates the location and view angle of each viewof the plurality of views.

The location/view angle indicator window can have a selection icon thatcan be selected to cause it to be displayed in the composite videoduring a given period of time of the timeline.

The metadata can enable at least one of a still camera image, animation,digital score board, digital scorebook, or live video to be synchronizedso that it can be displayed in the composite video during a given periodof time of the timeline.

The media player can be configured to create a multiple view video datapackage comprising a plurality of media files and metadata filescorresponding to multiple views of a recorded event.

The multiple view video data package may include a composite videoproduced using the plurality of media files and metadata files.

The media player can be configured to post the multiple view video datapackage to an internet media-sharing website thereby enabling users ofthe internet media-sharing website to at least one of comment on thecomposite video, rate the composite video, or download the plurality ofmedia files and the plurality of metadata files.

Another aspect of the present invention involves a method of mediasynchronization and collaboration, comprising recording an event using aplurality of media recording devices that independently record the eventto produce a plurality of recorded media data corresponding to aplurality of views of the event, storing media data and metadatacorresponding to the plurality of views of the event to a data storage,the metadata including time samples in accordance with a common timereference, and providing a media player comprising a processor and agraphical user interface that uses the metadata to synchronize and playthe plurality of views of the event.

The method may also include producing a composite video by selectingusing the graphical user interface a sequence of selected views of theplurality of views of the event that play during a periods of time aspart of an overall timeline of the composite video.

The may also include creating a multiple view video data packagecomprising a plurality of media files and a plurality of metadata filesused to produce the composite video.

The multiple view video data package may include the composite video.

The method may also include posting the multiple view video data packageto an internet media-sharing website thereby enabling users of theinternet media-sharing website to at least one of comment on thecomposite video, rate the composite video, or download the plurality ofmedia files and the plurality of metadata files.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described with reference to the accompanyingdrawings. In the drawings, like reference numbers indicate identical orfunctionally similar elements. Additionally, the left-most digit(s) of areference number identifies the drawing in which the reference numberfirst appears.

FIG. 1 depicts a user of a phone recording an event that is conveyed tothe cloud;

FIG. 2A depicts a first exemplary method in accordance with theinvention;

FIG. 2B depicts a second exemplary method in accordance with theinvention;

FIG. 2C depicts a third exemplary method in accordance with theinvention;

FIG. 2D depicts a fourth exemplary method in accordance with theinvention;

FIG. 3A depicts a fifth exemplary method in accordance with theinvention;

FIG. 3B depicts a sixth exemplary method in accordance with theinvention;

FIG. 3C depicts a seventh exemplary method in accordance with theinvention;

FIG. 3D depicts an eighth exemplary method in accordance with theinvention;

FIG. 4A depicts a ninth exemplary method in accordance with theinvention;

FIG. 4B depicts a tenth exemplary method in accordance with theinvention;

FIG. 4C depicts an eleventh exemplary method in accordance with theinvention;

FIG. 5A depicts a twelfth exemplary method in accordance with theinvention;

FIG. 5B depicts a thirteenth exemplary method in accordance with theinvention;

FIG. 5C depicts an fourteenth exemplary method in accordance with theinvention;

FIG. 6 depicts a fifteenth exemplary method in accordance with theinvention;

FIG. 7 depicts an exemplary computing architecture in accordance withthe invention;

FIG. 8 depicts an exemplary graphical user interface of a media playerfor playing a single view of an event using media and metadatacorresponding to the event in accordance with the invention;

FIG. 9 depicts an exemplary graphical user interface of a multiple viewmedia player for producing a composite video from media and metadatacorresponding to multiple views of an event in accordance with theinvention; and

FIG. 10 depicts an exemplary graphical user interface of a multiple viewmedia player for producing a composite video from media and metadatacorresponding to multiple views of a baseball game in accordance withthe invention.

DETAILED DESCRIPTION OF THE INVENTION

In accordance with one aspect of the present invention, a mediarecording device interfaced with a cloud computing environment (i.e.,the “cloud”) conveys a recorded media to the cloud in near real-timewhile the media is being recorded by the media recording device, wherethe conveying of the media to the cloud may be based upon the occurrenceof a user-defined event and/or a user command, where the media may ormay not be encrypted and storage of a local copy of the media isoptional. The media recording device may also upload to the cloud apreviously recorded media stored on the media recording device. Theuploading of the previously recorded media may be in accordance with aschedule and/or upon the occurrence of a user-defined event, where themedia may be encrypted when stored on the media recording device andsubsequently uploaded to the cloud or stored on the media recordingdevice as unencrypted data that is then encrypted as it is uploaded tothe cloud or after it has been uploaded to the cloud (i.e., by a cloudservice provider as opposed to encryption by the media recordingdevice). Rules may be established and enforced for forbidding access,modification and/or erasure from the recording device and/or the cloud.

Under one arrangement a first recording device forwards a media to asecond recording device that conveys the media to the cloud, where thefirst recording device could be a service interfacing with multiplesecond recording devices and which could be configured to havesubstantial network bandwidth significantly greater than that requiredand typically used by the individual second recording devices.

In accordance with another aspect of the invention, data latency rulescan be established to control whether various optional functionspertaining to processing of a recorded media (e.g., encryption, localstorage, adding of metadata, etc.) are performed or not or an amount orrate corresponding to a processing function being performed (e.g., datacompression level, data sampling rate, data buffering amount,resolution, etc.) for a given recorded media. Data latency rules may bebased on one or more established data latency thresholds correspondingto one or more user-defined conditions. For example, a data compressionrate and/or data sampling rate may be controlled in real-time asrequired to keep data latency below an established data latencythreshold. Moreover, a data latency threshold may be conditional in thatit might be modified or overridden, for example, based on an event suchas a sensed event. For example, given a sensed fire condition, allunnecessary processing may be avoided regardless of an established datalatency threshold.

Under one arrangement, a user may establish a set of parameters relatingto which media data processing functions are to be used or not used andthe extent to which they may be used (e.g., sampling rate, amount ofdata compression, data resolution) so as to control data latency whilemeeting certain user requirements (e.g., user always wants encryption),where media data processing functions are either turned on or off or anextent (e.g., rate, resolution) changed based on criteria or rulesestablished by a user. Such criteria may determine a sampling rate, anamount of data compression, the extent of which metadata is added. Forexample, a user may establish a parameter where video is to be capturedat a 1080i (i.e., 1920×1080) resolution but, when a certain criteria ismet, the resolution is to change to a different resolution, which maybe, for example, a lower resolution (e.g., 702p) or higher resolution(8640p).

Under another arrangement a command may be provided by the user of amedia recording device to change a mode of operation. For example, an“upload button” might be pressed or an upload voice command may bespoken to cause an upload to the cloud function to be startedimmediately. Similarly, a “fast upload button” might be pressed or afast upload function otherwise initiated to cause an upload function tobe initiated under conditions that provide for minimal data latency.Generally, a user may establish one or more events that correspond to‘fast load triggers’, whereby the occurrence of such an event causes thefast upload function to be initiated.

In accordance with one feature of the invention, a data latencyindicator may be provided which might be a number and/or a color or someother indicator.

In accordance with another feature of the invention, an event may resultin other applications running on a device (e.g., cell phone) beingturned off in order to speed up performance of the device or otherwisereduce data latency.

In accordance with yet another aspect of the invention, access controlscan be employed to prevent unauthorized access to or deletion of arecorded media stored on the media recording device and/or on the cloud.One or more deletion events can also be defined by a user, where a localcopy of recorded media will be automatically deleted from the mediarecording device based on the occurrence of a deletion event.

A media recording device can include a video recording device and/or anaudio recording device, for example, a camera and a microphone of amobile phone or a Bluetooth or Wi-Fi connected device, and the media canbe, for example, video (still picture or movie) and/or audio datarecorded by the video and audio recording devices of the phone, wherethe recorded media is in the form of digital data. Other examples ofmedia recording devices include a media recording device located in ahome or business, a media recording device (e.g., dash cam) located in avehicle (e.g., car, truck, emergency vehicle, plane, helicopter, drone,etc.), a media recording device (e.g., body cam) associated with aperson or animal, or a media recording devices associated with a fixedobject (e.g., bridge, tree, light post, gas pump, etc.). Recorded mediamay be in the form of text, for example, where an audio recording deviceconverts voice to text. Video data may correspond to a picture or videotaken from the front (display side) of a cell phone or from the back ofa cell phone, or both, which may be taken simultaneously. As such, anevent being filmed may be captured at the same time the user of a mediarecording device is captured (e.g., a selfie).

Encryption of media data, whether stored on the media recording deviceor on the cloud, may for example involve use of a symmetric keyencryption scheme or a public key encryption scheme. Encryption may beperformed by the media recording device such that a local copy can bestored in encrypted form and/or media is conveyed to the cloud inencrypted form. Alternatively, media may be conveyed in unencrypted formand then encrypted by a cloud service provider as it is being receivedand stored on the cloud or sometime after it has been received andstored on the cloud.

Various data access and user authentication methods can be employed tocontrol access and/or deletion to data stored on the media recordingdevice or on the cloud. Such methods may include a password, asignature, or a biometric such as an eye scan, a fingerprint scan, afacial image (i.e., individual's photo, a selfie), a recognized voice,or the like. At least one physical key (e.g., a dongle) may be requiredto access data, where multiple keys (or dongles) distributed to multiplepersons may be required to access or delete data. A third partyauthentication service provider might be used such as VeriSign.Generally, one skilled in the art of protecting data stored on anelectronic device or across a network will understand that all sorts ofmethods can be employed to control access to data and to authenticate auser, where such controls can also be used to prevent unauthorized datadeletion. Moreover, rules can be employed in conjunction with suchaccess control methods, for example, access and/or data deletion may belimited to a certain time period, require a certain aging of data (i.e.,an elapsed period of time), require an event to have occurred (such asdescribed below), require the media recording device to be in a certainlocation, etc. Generally, the concepts described below relating touser-defined events being used to determine the starting and stopping ofrecording of a media recording device and corresponding conveying ofrecorded media to the cloud or the uploading of previously recordedmedia data to the cloud can also be applied for controlling access toand deletion of media stored on the cloud or on the media recordingdevice. Alternatively, it may be desirable that a media can beidentified that cannot be deleted from the media recording device and/orfrom the cloud under any circumstance or without participation by athird party given control over such access/deletion such as an attorney,an editor of a publication, or some third party service.

Under a first arrangement, which is depicted in FIG. 1, a user 100 of amedia recording device 102 runs an application (or ‘app’) or otherwiseselects a mode of operation of the media recording device 102 thatcauses a media 104 to be conveyed to the cloud 106 while the media 104is being recorded by the media recording device 102 upon the occurrenceof a user command. The media may be encrypted by the media recordingdevice prior to the media being conveyed to the cloud. Alternatively,the media may be conveyed to cloud in unencrypted form where it then mayor may not be encrypted. Under this arrangement, which may be referredto as a user-activated near real-time mode, storage of a local copy ofthe media on the media recording device is optional, where the media maybe conveyed directly to the cloud and may never actually be storedlocally on the media recording device. For example, a user of a mobilephone such as an HTC® phone may record an event with their phone and themedia data would be conveyed directly to the cloud as it was beingrecorded with very low latency between the time a given data packet isrecorded until it is stored on the cloud, where the data correspondingto the event would not be stored or otherwise be present on the phoneunless it is necessary to temporarily buffer data for some requiredreason, for example, due to a poor or non-existent data connectionbetween a first and second media recording device or between a mediarecording device and the cloud. Alternatively, the user may choose tostore a local copy of the media data on the phone while also conveyingthe media data directly to the cloud as the media is being recorded,where the local copy of the media data may be encrypted or unencryptedand where the data conveyed to the cloud may be encrypted prior to beingconveyed to the cloud or the data may be encrypted after it has beenconveyed to the cloud. For example, unencrypted data might be conveyedto a cloud service provider that encrypts the data it receives from themedia recording device prior to storing it, where the time required toencrypt the media would not add to the data latency of the mediarecording device, but the data would be vulnerable while be conveyed tothe cloud because it is in an unencrypted form. Depending on theconditions of a given situation, it may be preferable to reduce datalatency of the media recording device by not requiring the mediarecording device to encrypt a recorded media prior to conveying it tothe cloud. As such, a user can establish rules used to control whethersuch processing is performed or not depending on the conditions of agiven situation. For example, a user might set up a rule whereby videofootage of a business security system would be automatically encryptedand conveyed to the cloud upon the occurrence of a user-defined securityalarm event but encryption should not be performed should a firecondition (e.g., fire alarm, a sensor detecting smoke or heat, etc.)also be detected. Generally, all sorts of rules can be employed tocontrol the processing performed prior to conveyance of media data tothe cloud based on one or more conditions so as to control data latency.

FIGS. 2A-2D depict exemplary methods corresponding to a user-activatednear real-time mode, where the amount of data latency between the time agiven data packet is recorded until it is stored on the cloud depends onwhether or not media is encrypted and whether or not a local copy of themedia (or encrypted media) is stored on the media recording device. Inreference to FIG. 2A, one skilled in the art will recognize that oncemedia has been received by the cloud it may be recorded by a cloudservice provider prior to storage or at any time thereafter.

Under a second arrangement, a media recording device is configured toautomatically begin and/or stop recording media to be encrypted andconveyed to the cloud while the media is being recorded by a mediarecording device upon the occurrence of a defined event or events. Underthis arrangement, which may be referred to as an event-activated nearreal-time mode, storage of a local copy of the media on the mediarecording device is optional, where the local copy of the media data maybe encrypted or unencrypted while stored on the media recording device.

An event can be generally described as an occurrence that meets anestablished criterion, condition or rule that can be recognized by acontrol system (e.g., an application running on a cell phone). Forexample, an event may be based on a position of anobject/person/animal/vehicle, for example, a cell phone's specificlocation (i.e., latitude, longitude) as might be determined by alocation system such as a global positioning system (GPS). An eventmight correspond to a status of a media recording device, for example, abattery status or a signal strength status. Similarly, an event might bebased on a position of an object relative to a location of anotherobject, where both objects might be fixed or mobile. Such location basedevents are commonly known as geolocation events where, generally, anevent can be defined based on the location of one or more objectsrelative to one or more defined areas (e.g., a perimeter or a propertyor a building, or a room within a building) corresponding to one or morelocations. An event may be based on movement, lack of movement, or achange in movement (e.g., speed or direction) of an object, which mightbe detected using a compass. For example, an ‘impact threshold’ may beestablished corresponding to an abrupt movement change indicating animpact associated with a media recording device (e.g., hitting theground, being in a vehicle crash, etc.).

An event may be based on a position, for example, a position of thephone within a coordinate system. An event may relate to a movement ofthe phone or the non-movement of the phone, which might be detectedusing an accelerometer. An event may relate to a detected movement,which might be detected by a motion or proximity detector/radar. Anevent may be based on an orientation of an object, which might bemeasured using a 6-DOF measurement device. In one arrangement, theorientation of a phone may be determined using a magnetometer containedin a media recording device.

An event may be based upon an emergency or alarm situation, which mightinvolve a severe weather advisory or warning relating to a thunderstorm,tornado, hurricane, snowstorm, high wind, etc. or any other sort ofemergency situation such as a vehicle accident or crash, a break-in, afire, a flood, a landslide, a prisoner escape, a riot, a hazardousmaterials spill, a runaway train, an airplane experiencing an emergencysituation, etc. For example, a media recording device may be set toautomatically begin recording if a nearby nuclear reactor alarm were tosound or if a person presses a medical alert button.

An event might involve a government controlled security level, forexample, a Transportation Security Administration (TSA) security level,Home Security level, or DEFCON level.

An event might be a sensed environmental condition such as a sensedtemperature, humidity, light, smoke, carbon dioxide, seismic event,sound intensity and/or frequency, pressure, altitude, water depth, orthe like, which might be measured using one or more sensors. Forexample, a sensor might detect an earthquake, an explosion, thunder, agunshot, or a scream. Similarly, an event might be a sensed physicalcondition of a person or animal such as a heart rate, breathing rate,skin resistivity, blood pressure, body temperature, blood sugar level,etc. One skilled in the art will also recognize that if the mediarecording device is configured to received sensed information then themedia recording device can also perform various other processing beyonduploading media to the cloud relating to the sensed information. Forexample, seismic information sensed by the media recording device mightbe used to identify the location, timing, and magnitude of a seismicevent, which might even be used to determine an amount of time before acatastrophic even will occur at the location of the recording mediadevice for providing warning, instructions, or other relevantinformation.

An event may involve the recognition of a command such as a voicecommand, a hand gesture, or a RF signal command.

FIGS. 3A-3D depict exemplary methods corresponding to an event-activatednear real-time mode, where the amount of data latency between the time agiven data packet is recorded until it is stored on the cloud depends onwhether or not media is encrypted and whether or not a local copy of themedia (or encrypted media) is stored on the media recording device. Inreference to FIG. 3A, one skilled in the art will recognize that oncemedia has been received by the cloud it may be recorded by a cloudservice provider prior to storage or at any time thereafter.

Under the first or second arrangements, efforts can be made to limitlatency between recording and conveyance to the cloud to an amount lessthan or equal to a defined latency limit, where to meet an establishedlatency limit, storing of a local copy of the media and/or encryption ofthe local copy may not occur.

Under a third arrangement, a local copy of a recorded media is stored onthe media recording device and all or part of the stored local copy ofthe media is encrypted and conveyed to the cloud in accordance with adefined schedule. The scheduled uploading of the media to the cloud maybe referred to as a scheduled upload mode, where the local copy of themedia data may be encrypted or unencrypted while stored on the mediarecording device. Optionally, the media data can be automaticallydeleted from the media recording device once it has been conveyed to thecloud. For example, a user of a phone may choose to have media datamoved from a phone to the cloud weekly, daily, hourly, or at specificscheduled times and/or in response to an event, for example, a voicecommand, a location, an emergency condition, etc. The media may bestored and uploaded using a rolling period of time, for example, at theend of each day, the stored media data from the same day one week priormay be automatically deleted such that, at any given time, there isstored media data for the most recent seven days, where the rollingperiodic upload mode can be overridden, the rolling period of time canbe changed (increased or decreased), and specific subsets of storedmedia data can be identified as not to be deleted. For example, arolling period of time mode might be configured to only delete mediadata from 12 pm to 6 am or to never delete data recorded on a Saturday.Generally, one skilled in the art will recognize that all sorts ofoptions for controlling one or more periods of time where stored mediadata would be uploaded to the cloud and automatically deleted from localstorage, or not, are possible.

FIGS. 4A-4C depict exemplary methods corresponding to a scheduled updatemode, where the amount of data latency between the time uploading of astored media begins until it is stored on the cloud depends on whetheror not the media is encrypted by the media recording device. Inreference to FIG. 4A, one skilled in the art will recognize that oncemedia has been received by the cloud it may be recorded by a cloudservice provider prior to storage or at any time thereafter.

Under a fourth arrangement, a local copy of a recorded media is storedon a media recording device and then later automatically conveyed to acloud computing environment upon the occurrence of an event, such asdescribed above, where the media may be encrypted prior to beingconveyed to the cloud and where the local copy of the media data may beencrypted or unencrypted while stored on the media recording device.Optionally, the media data can be automatically deleted from the mediarecording device once it has been conveyed to the cloud.

FIGS. 5A-5C depict exemplary methods corresponding to an event-activatedupdate mode, where the amount of data latency between the time uploadingof a stored media begins until it is stored on the cloud depends onwhether or not the media is encrypted by the media recording device. Inreference to FIG. 5A, one skilled in the art will recognize that oncemedia has been received by the cloud it may be encrypted by a cloudservice provider prior to storage or at any time thereafter.

Under a fifth arrangement, a local copy of a recorded media is deletedupon the occurrence of an event, such as described above. FIG. 6 depictsan exemplary method corresponding to an event-activated delete mode.

Various metadata can be conveyed to the cloud along with the media datasuch as the media author, media title, date and time of the mediarecording, location and/or orientation of the media recording device,velocity, acceleration, temperature, barometric pressure, biometricdata, light levels, etc. Metadata might include the person or persons ina video, or a short description or keyword(s) such as wedding, pet'sname, flowers, waterfall, food, or the like. Generally, one skilled inthe art will understand that such metadata can be used to enableprocessing of the media data from the cloud to include a user retrievinga subset or subsets of such media data based upon a query of themetadata stored along with the media data. Whether or not metadata isadded to media prior to it being conveyed to the cloud can also becontrolled in accordance with a data latency limit in a manner similarto how encryption can be controlled. Similarly, whether or not metadatais added to media prior to it being conveyed to the cloud can also becontrolled in accordance with an established rule and one or moreconditions of a situation. Under one arrangement, metadata (e.g., timingand location) corresponding to one or more records corresponding to oneor more recorded media of one or more recorded media devices may be usedto located the source of a sound as recorded by the one or more mediarecording devices, where one skilled in the art will recognize that onerecorded sound might provide a range of the source relative to arecording device, two recordings of the sound may determine a planerelative to the locations of the recording devices, and three recordingsof the sound may identify a coordinate of the source relative to thethree recording devices. A source of a sound could be, for example, agun or a tornado. One skilled in the art will recognize that all sortsof data processing involving multiple recorded media data by one or moremedia recording devices are possible.

A user interface could be used to define the events, rules, andconditions required to support event-activated approaches describedherein. Similarly, a user interface can be used to define limits such aslatency limits, to manage encryption, and to enter metadata. Similarly,a user interface can be used to produce queries used to retrieve mediadata from the cloud. Generally, one skilled in the art of datamanagement will understand that such interfaces can be employed topractice the invention.

The following are examples of the use of the present invention:

-   -   a baby monitoring system whereby a sensed condition such as a        temperature, irregular heartbeat, or the like cause the baby        monitoring system to convey sound, video, and sensor information        to the cloud, where it may then be forwarded to medical        personnel either manually or automatically as part of a service.    -   a business security system whereby a detected forced entry        automatically conveys security footage to the cloud, where it        may then be forwarded to security personnel or police either        manually or automatically as part of a service.    -   a structure health monitoring system whereby upon the occurrence        of an earthquake sensor information pertaining to the health of        the structure (e.g., bridge, building, dam, etc.) is        automatically conveyed to the cloud where video footage may also        be automatically recorded and conveyed.    -   a personal video monitoring system worn on a person        automatically beginning recording and conveying video and/or        sensor information to the cloud upon recognition of an irregular        heartbeat or other sensed characteristic of a person, which        might result from fear, an accident, a medical condition (e.g.,        a diabetic seizure), excitement, or the like.    -   a vehicle monitoring system that begins recording occurrences        outside the vehicle and/or inside the vehicle given a sensed        condition such as a break-in of the vehicle, the vehicle being        in an accident, the vehicle being driven recklessly, etc.    -   a drone-based video surveillance system reacts to an occurrence        on the ground (e.g., a detected explosion) by directing the        drone to reduce altitude or alter course so as to achieve a        different surveillance location or to cause a zoom function in a        camera to zoom in so as to better view the occurrence on the        ground.

The present invention can be practiced using publicly availablecomputing devices, communications networks, and related software or canbe practiced using proprietary computing devices, communicationsnetworks, and/or software. Rules and thresholds and the like can beestablished for one or more media recording devices using one or morecomputing device (e.g., a desktop computer) other than a recordingdevice. Similarly, an interface can be provided to access media datastored on the cloud via computing devices other than a media recordingdevice. Under one arrangement, a product is provided that includes asoftware application resident on a media recording device and a softwareapplication resident on a computing device other than a recordingdevice. For example, an application running on a cell phone may storemedia data to the cloud that is later accessed via a desktop computervia an internet connection. Similarly, an application (e.g., adashboard) executing on a desktop computer may be used to configureparameters (e.g., rules, thresholds, etc.) relating to a user accountthat are then loaded by a cell phone application and used to manage theconveyance of recorded media data to the cloud by the cell phone. Underanother arrangement, one or more other applications used to manageevents such as a calendar management application (e.g., MicrosoftOutlook®) can be used to establish and manage events that are used tomanage the conveyance of media data to the cloud. For example, a meetingrequest received via an email may establish a location and a time usedin a rule used to manage the conveyance of media data to the cloud.Similarly, an alert condition established in a weather alert applicationmight be inherited by another application managing the conveyance ofmedia data to the cloud. Under yet another arrangement, the applicationmanaging the conveyance of media data to the cloud may interface withone or more publicly available data sources (e.g., National WeatherService, USGS Earthquake Early Warning system, a RSS news blog) and/orprivate data sources (e.g., a Social Directory API), where data providedby the one or more publicly available data sources and/or private datasources may be used, for example, to determine the occurrence of anevent.

The present invention may be used as part of a monitoring service wherethe control of media recording functions can be at least partiallymanaged by the monitoring service. For example, one or more mediarecording devices within a home or business may be activated based on adetected condition, a schedule, or as part of a random status check,where certain parameters are controllable by a user (e.g., home owner,business owner).

The present invention may take advantage of artificial intelligencealgorithms that enable a media recording device to establish its ownrules make its own decisions regarding which functions should beemployed and to what extent as determined based on one or more events.

Under one arrangement, a media recording device may be configured tooperate without displaying images being recorded on the display of thedevice. For example, a phone may be filming and streaming a video to thecloud without displaying the video on the display of the phone. This‘non-display recording mode’ might be activated by a user selecting abutton, providing a voice command, or automatically due to theoccurrence of an event (e.g., a detected abrupt movement), etc. in thesame manner as media recording can be activated.

In accordance with an embodiment of the invention, multiple mediarecording devices may be configured to collaborate. In one arrangement,data samples from different devices can have relative timings that arecoordinated. For example, if three media recording devices are recordingthe same event from three different locations the their data samples mayhave staggered timing such that the timing of data samples from thefirst media recording device may be provided every 3 seconds beginningat time t₀, data samples from the second media recording device may beprovided every 3 seconds beginning at time t₀+1 second, and data samplesfrom the third media recording device may be provided every 3 secondsbeginning at time t₀+2 second. Alternatively, two or more mediarecording devices may coordinate their times such that they take datasamples at substantially the same times, t₀, t₁, t₂, etc. Other forms ofcollaboration include: 1) sharing of rules, thresholds, sensorinformation, and the like such that a set of parameters can beestablished that is used to manage conveyance of media data produced bymultiple media recording devices, 2) sharing of recorded data amongmedia recording devices allowing the display of information frommultiple devices as an event is being recorded by the devices, 3)sharing of warnings and messages between devices upon occurrences ofevents (e.g., Fred's phone is code red and located at x,y,zcoordinates).

FIG. 7 depicts an exemplary computing architecture 700 in accordancewith the invention. Referring to FIG. 7, the computing architecture 700includes a plug-in application 702 (e.g., a cell phone ‘app’) and a PCapplication 704. The plug-in application 702 includes a front end 706and an application program interface (API) 708. The front end 706provides a user interface that enables a user to log into theapplication which would involve use of a user management service 712,which provides user login and authentication capabilities, andcorresponding payment management service 714, which provides for variousmeans of payment for the application's services, products, etc. The usermanagement service 712 provides notification to the API 708 that theuser is authorized (i.e., authenticated and appropriates payments havebeen made) to use the application. The API 708 interfaces with a cloudservice 720 and interfaces with a media server service 716 which streamsmedia data to one or more media file databases 718, which may beprovided by a cloud service or by another data storage service 720. TheAPI 708 also provides metadata associated with the media data to one ormore metadata file databases 719. The API 708 also interfaces with a webserver service 722, which interfaces with a website 724 and a dashboard726. The website 724 provides unregistered users a product description,pricing, and a product registration form. Once registered, a user isable to access the dashboard 726 that can be used to configureparameters relating to the user's account and provides access to theuser's media files.

In accordance with another aspect of the present invention, a pluralityof media recording devices interfaced with a cloud computing environmentconvey a corresponding plurality of recorded media and metadata to thecloud computing environment, where the plurality of recorded media andmetadata may or may not correspond to multiple views of the samerecorded event, and where the metadata enables the plurality of recordedmedia to by synchronized. The metadata includes time samples inaccordance with a common time reference (i.e., a universal clock, atomicclock, NIST, US Naval Observatory clock, etc.) and may includemagnetometer data that can be used to determine the view angle of amedia recording device (e.g., a smartphone) and/or location informationfrom a location system such as GPS location information. One skilled inthe art will recognize that the plurality of media recording devices mayrecord media and store metadata without using the cloud.

The present invention enables multiple individuals to use multiple mediarecording devices, for example, smartphones to independently andasynchronously record an event (e.g., a baseball game) from different adhoc locations, where the beginning and ending recording times of thecorresponding multiple recorded views of the event are determined by theindividuals recording the different view (or media clip) of the event.Generally, the multiple individuals recording an event are not requiredto collaborate prior or during their recording of the event and a givenindividual may not even have any a priori knowledge of the recording byother individuals or even be aware of the recording of the event by anyother individuals. A recorded event can be something involving somenumber of people that are collocated such as those attending a weddingwhere multiple persons might use their smartphones to record all or partof the wedding from different locations. Or, a recorded event mightinvolve multitudes of people at many different locations, for example,persons recorded celebrating the occurrence of a new year. An eventmight also be some sort of tragedy such as a natural disaster (e.g., anearthquake), a terrorist attack (e.g., 911), a volcanic eruption, or thelike.

The metadata stored during the recording of the multiple recorded viewsby the multiple media recording devices allows the various recordedmedia clips to then be synchronized such that multiple views of an eventcan be used collaboratively to produce one or more composite videos thatare a combination of a sequence of different views of an event takenduring periods of time. Generally, many different types of recordingdevices (e.g., video, cameras, audio recorders, etc.) can independentlyand asynchronously record an event, where the devices store metadataalong with their recordings in accordance with the invention, and therecorded output of the devices can be later synchronized and usedtogether in a collaborative matter, for example, a composite mediaoutput can be produced.

Referring again to the architecture of FIG. 7, the plug-in application702 running on the multiple smartphones of multiple individuals is usedto record multiple views of an event where the media and metadatacorresponding to the multiple views is stored in a media database 718and metadata database 719 provided by a cloud service or another datastorage service 720. A PC application 704 (aka. a dashboard application)of any one of the multiple individuals can be provided the media andmetadata of the multiple views of the event where the views can besynchronized and combined in various ways to produce various compositevideos made up of different views of the event recorded by the multiplesmartphones.

FIG. 8 depicts an exemplary graphical user interface (GUI) 800 of amedia player of a PC application 704 for playing a single view of anevent using media and metadata corresponding to the event. Referring toFIG. 8, the GUI includes a media player viewing window 802 and alocation/view angle indicator window 804, which would typically displaya coordinate system, a floor plan, a map or any other of various typesof locational information that would provide an indication of thelocation 806 of a media recording device and also the view angle 808 ofthe media recording device which is indicated by a cone. Thelocation/view angle indicator window 804 may also include various othertypes of information such as weather radar, satellite imagery, and thelike. The GUI includes an information window 810, which can be used fordisplaying the product name, logos, a banner, advertisements, and otherinformation. The media player viewing window 802 is controlled usingmedia player controls 812, which typically includes a current framecontrol 814 shown on a timeline 816 as well as various other controlbuttons for controlling basic media player functions (e.g., pause/play,fast forward, time, closed caption, full screen, sound control, zoom,etc.).

The location/view angle indicator window is shown as a plan view butcould alternatively be presented using three dimensional viewingtechniques to as to indicate view elevation. View elevation might alsobe indicated using a numerical indication.

FIG. 9 depicts an exemplary GUI 900 of a multiple view media player of aPC application 704 for producing a composite video from media andmetadata corresponding to multiple views of an event. Referring to FIG.9, the GUI 900 includes a media player viewing window 802, alocation/view angle indicator window 804, an information window 810, andmedia player controls 812 like the GUI 800 of FIG. 8. As seen in thelocation/view angle indicator window 804, the locations 806 a-806 d ofmultiple media recording devices are indicated along with theirrespective view angles as indicated by cones 808 a-808 d. Fourmulti-view media player viewing windows 902 a-902 d are used to displaythe multiple views of the event as recorded by the multiple mediarecording devices having the locations 806 a-806 d and viewing angles808 a-808 d. The multi-view media player viewing windows 902 a-902 deach have corresponding selection icons 904 a-904 d, which in apreferred embodiment are color-coded, and view mode control buttons 906a-906 d for controlling the viewing mode of each respective viewingwindow 902 a-902 d. The view mode control buttons 906 a-906 d are usedto select which view is being played during a given period of time aspart of the overall timeline of a composite video, where essentially thecomposite video consists of a sequence of selected views. An additionalselection icon 904 e is shown in the location/view angle indicatorwindow 804, which allows the window 804 to be selected and included in acomposite video. In a preferred embodiment the timeline 816 of thecomposite video is color-coded consistent with the color coding of theselection icons 904 a-904 d corresponding to the views selected for thedifferent portions of the composite video. Viewing modes may include mayinclude, for example, OFF, PLAY, and PERIODIC FRAME modes, where thePLAY mode the media normally and the PERIODIC FRAME mode samples framesfrom the media data periodically (e.g., once per second) and displaysthe same frame for a period of time. Generally, the PERIODIC FRAME modereduces bandwidth requirements but also reduces overall movement in theGUI, which might be preferable to certain users when selecting amongviews. Views may be played simultaneously or may be played one at atime. Non-selected views may update frames periodically (e.g., once persecond) whereas the selected view plays normally.

To the right of the media player viewing window 802 is a view controlwindow 908, which may include various functions specific to the multipleviews or to the composite video. For example, a speed control 910 couldbe used to vary the speed (e.g., slow motion) at which one or more viewsis playing or a multi-view combination control 912 might be used, whichcauses a sequence of different views of the same or otherwiseoverlapping periods of time (e.g., four view angles of a bat hitting thewinning home run). Generally, different types of view editing functionscan be included in the view control window 908.

In accordance with another aspect of the invention, metadata files canbe used by a plurality of media service providers to enable their mediaproducts to be synchronized, where such media products may involvevideos, still camera images, animations, GUIs, etc. As such, differentvendor products can be used to produce composite products. For example,a provider of digital score board displays for sporting events couldprovide metadata files with its display data files enabling the displaysto be integrated with other products that could benefit from thedisplays.

FIG. 10 depicts an exemplary GUI 1000 of a multiple view media player ofa PC application 704 for producing a composite video from media andmetadata corresponding to multiple views of a baseball game, whichincludes products that might be provided by other service providers. Forexample, the information window 804 of the GUI 1000 might depict abaseball park 1002, which might be a live video image or images, adigital scoreboard 1004, and a digital scorebook 1006, which might beprovided by other service providers but integrated into the GUI andsynchronized with the multiple views of the baseball game using themetadata provided by the various service providers. As shown, selectionicons 904 e-904 g can be associated with the output of these third partyproducts, which causes their images to be included in a composite video.

Generally, the present invention enables various types of specializedtheme-based PC applications 704 having GUIs that are tailored forspecific types of events such as sporting events (e.g., tennis,basketball, soccer, baseball, football, volleyball, etc.), weddings,concerts, plays, recitals, political events, and any other type of eventwhere specialized products can be integrated as a result of productvendors complying with metadata requirements that enable synchronizationof products. Such specialized PC applications 704 can have advertisingfocused on specific groups that would be interested in the specific typeof events. For example, advertisements targeted at baseball fans mightbe displayed on a GUI having a baseball theme.

In accordance with a still further aspect of the invention, a multipleview video data package comprising a plurality of media files andmetadata files corresponding to multiple views of a recorded event canbe posted to an internet media-sharing website (e.g., YouTube®) ofFaceBook®), where the package may or may not include one or morecomposite videos produced using the plurality of media files andmetadata files. As such, other users of the media-sharing website cancomment on and rate (e.g., thumbs up or down) a composite video and/orcan download the media files and metadata files corresponding to therecorded event to their own PC applications 704 allowing them to producetheir own composite video(s). Alternatively, the internet media-sharingwebsite might provide a real-time dashboard application for producing acomposite video using the media files and metadata files.

As such, the present invention enables a new collaborative media sharingenvironment whereby users can combine views taken from their smartphoneswith other views or otherwise participate in the creation of compositevideos from whatever views of an event are available. Users can posttheir multiple view video data packages and/or media and metadata for asingle view using an identifier such as a Hashtag#.

Moreover, various other social media products can be leveraged includingTwitter®, which might be used to organize views in real time (e.g.,someone needs to move to the right of the stage to get a view . . . ).Similarly, applications such as LinkedIn® might be used to organize theviews recorded by different groups or organizations.

In accordance with an aspect of the invention, information may bedisplayed at an event that can be captured with a media recording deviceand be used to coordinate a collaborative recording of media relating toan event. For example, a barcode (or QR code) can be displayed on aplacard, on a display, or in a program for an event that a user of asmartphone can use the camera of the smartphone to take a picture of thebarcode, where a software application running on the phone woulddecipher the barcode information and use it to coordinate thecollaborative recording of media relating to the event. Under onearrangement, a person organizing the collaborative recording of mediarelating to an event can provide information to a graphical userinterface of a software application that subsequently can create abarcode (or QR code), which can then be displayed at an event such thatit can be captured by media recording devices of persons wanting toparticipate in the collaborative recording of media relating to theevent.

To support synchronization, digital watermarks might be placed oncertain frames during the recording of an event and a dashboardapplication might vary relative time bases of different views using thedigital watermarks.

In accordance with a further aspect of the invention, the media andmetadata corresponding to a given recorded event can be made publiclyavailable or access to such media and metadata can be controlled suchthat it is only provided to one or more authorized persons. Under onearrangement, a service provider may convey media and metadata toauthorized users in accordance with access control permissions which mayadministered using one or more layers of administration that controlmembership in groups where access to media and metadata may involve anaccess control list, a password, and/or a role. Generally, control ofaccess to media and metadata can be controlled in the same or similarway that access to location based information is described in U.S. Pat.No. 7,525,425, which is incorporated by reference herein in itsentirety. More particularly, a person who records an event can beconsidered the owner of the corresponding recorded media and metadataand can determine which other person or persons have access to suchdata. For example, an owner of certain media and metadata may restrictaccess to persons included on an access control list, persons that knowa password, or persons provided access because of a role (e.g., adetective, an administrator, etc.). Alternatively, a person may requestan owner to provide access to certain media and metadata, which might begranted electronically, for example, by an owner responding to anautomated permission message in a manner similar to how a LinkedIn®connection can be requested and accepted.

In accordance with yet another aspect of the invention, a mediarecording device may include or be interfaced to a local media server ora data buffering device which might be a memory card, thumb drive, orthe like that enables media data and metadata to be stored periodically,for example, during a loss of a data link to the cloud. Under onearrangement, data may be automatically buffered to a local media serveror a data buffering device until successfully conveyed to the cloud inwhat can be described as store and forward streaming to the cloud. Underanother arrangement buffering to a local media server or a bufferingdevice only occurs when a link to the cloud is unavailable.

While particular embodiments of the invention have been described, itwill be understood, however, that the invention is not limited thereto,since modifications may be made by those skilled in the art,particularly in light of the foregoing teachings.

1. A system for media synchronization and collaboration, comprising: adata storage; a plurality of media recording devices used by a pluralityof users to independently and asynchronously record an event frommultiple locations thereby producing a plurality of recorded media datacorresponding to a plurality of views of said event, each of saidplurality of media recording devices conveying to said data storagemedia data and metadata corresponding to their respective view of saidevent, said metadata including time samples in accordance with a commontime reference; and a media player comprising a processor and agraphical user interface, said media player using said metadata tosynchronize and play said plurality of views of said event.
 2. Thesystem of claim 1, wherein said plurality of media recording devices isa plurality of smart phones.
 3. The system of claim 1, wherein saidgraphical user interface comprises a plurality of viewing windows usedto display the plurality of views of the event.
 4. The system of claim3, wherein each of said plurality of viewing windows has a correspondingselection icon of a plurality of selection icons.
 5. The system of claim3, wherein said plurality of selection icons are color-coded.
 6. Thesystem of claim 5, wherein said plurality of selection icons can be usedto select views of said plurality of views of said event to be playingduring periods of time as part of an overall timeline of a compositevideo that consists of a sequence of selected views.
 7. The system ofclaim 6, wherein said graphical user interface plays said compositevideo.
 8. The system of claim 6, wherein said timeline is color-coded toidentify the view of said plurality of views that is playing during agiven period of time of said timeline.
 9. The system of claim 6, wheresaid graphical user interface includes a multi-view combination controlthat causes a sequence of different views of the same time period oroverlapping time periods to be played in said composite video.
 10. Thesystem of claim 6, wherein said metadata includes location and viewangle data and said graphical user interface provides the ability todisplay a location/view angle indicator window that indicates thelocation and view angle of each view of said plurality of views.
 11. Thesystem of claim 10, wherein said location/view angle indicator windowhas a selection icon that can be selected to cause it to be displayed insaid composite video during a given period of time of said timeline. 12.The system of claim 6, wherein said metadata enables at least one of astill camera image, animation, digital score board, digital scorebook,or live video to be synchronized so that it can be displayed in saidcomposite video during a given period of time of said timeline.
 13. Thesystem of claim 1, wherein said media player is configured to create amultiple view video data package comprising a plurality of media filesand metadata files corresponding to multiple views of a recorded event.14. The system of claim 13, wherein said multiple view video datapackage further comprises a composite video produced using saidplurality of media files and metadata files.
 15. The system of claim 14,wherein said media player is configured to post said multiple view videodata package to an internet media-sharing website thereby enabling usersof said internet media-sharing website to at least one of comment onsaid composite video, rate said composite video, or download saidplurality of media files and said plurality of metadata files.
 16. Amethod of media synchronization and collaboration, comprising: recordingan event using a plurality of media recording devices that independentlyand asynchronously record said event to produce a plurality of recordedmedia clips corresponding to a plurality of views of said event; storingmedia data and metadata corresponding to said plurality of views of saidevent to a data storage, said metadata including time samples inaccordance with a common time reference; and providing a media playercomprising a processor and a graphical user interface that uses saidmetadata to synchronize and play said plurality of views of said event.17. The method of claim 16, further comprising: producing a compositevideo by selecting using said graphical user interface a sequence ofselected views of said plurality of views of said event that play duringa periods of time as part of an overall timeline of said compositevideo.
 18. The method of claim 17, further comprising: creating amultiple view video data package comprising a plurality of media filesand a plurality of metadata files used to produce said composite video.19. The method of claim 18, wherein said multiple view video datapackage further comprises said composite video.
 20. The method of claim19, further comprising: posting said multiple view video data package toan internet media-sharing website thereby enabling users of saidinternet media-sharing website to at least one of comment on saidcomposite video, rate said composite video, or download said pluralityof media files and said plurality of metadata files.